Learning with Locally Linear Feature Regularization

Machine learning models are successfully being used for problems in language, vision, and biology that have millions or tens of millions of features. A common approach to alleviating the complexity of high dimensional feature spaces is to penalize the L1 or L2 norm of the parameter vector. We may be able to design more effective regularizers, though, if we possess external information about which features should behave similarly. For example, word co-occurrence statistics or thesauri such as Wordnet can indicate similarities which are useful for predicting the topic or sentiment of a document, and biological databases of gene pathways give similarities of genes which can predict disease based on gene expression levels. We present a simple framework in which similarities between features are encoded as a graph on features and a regression model is learned whose feature coefficients are similar for neighboring nodes. Our regularization criterion is closely related to locally linear embedding, a method for learning low dimensional embeddings of unlabeled, highdimensional data [1]. Because of this, we name it locally linear feature regularization (LLFR).